The obvious solution to thieves accessing data on stolen Linux servers is LUKS.
LUKS lets you encrypt partitions so that they can only be accessed once unlocked with a passphrase.
Problem:
LUKS requires a passphrase to be typed-in to gain access to encrypted data (or even to boot, if the root partition is encrypted).
This is problematic in the case of unexpected reboots, regardless of cause.
/etc/crypttab used to allow a script to provide the passphrase, but systemd has a bug/feature which prevents this from working.
/etc/crypttab does allow a passphrase to be read from disk, but that appears pointless as it would need to be on unencrypted storage.
Solution:
Remotely Keyed LUKS would consist of two parts, remkeyfs and remkeyserver.
remkeyfs would be a remotely keyed encrypted filesystem.
Encrypted data (passphrases) would be stored locally, with unencrypted filenames.
The data would be encrypted with a key retrieved from remkeyservers via HTTPS.
/etc/crypttab would then contain references to /remkeyfs-mountpoint/volume-guid.passphrase
Passphrases stored in remkeyfs would be randomly-generated second-slot LUKS passphrases, allowing the original human-memorable passphrases to still be used, if ever needed.
The remkeyservers, which would be located on remote sites, would only serve the key to requests from the expected (probably NAT’ed) IP address.
When thieves power-up stolen servers, remkeyfs would be denied the key by the remkeyservers (due to the unexpected source IP address) and so remkeyfs would not decrypt the needed passphrase(s).
The stolen data would remain encrypted.
If the stolen servers had internet access, the failed attempt would be logged, including the source IP.
F.A.Q.
Q. Why store encrypted passphrases locally, rather than store unencrypted passphrases on the remkeyservers?
A. The remkeyservers would be a high value target if they contained all of the LUKS keys.
Also, this approach allows the remkeyservers to be run by third parties with whom you would not trust passphrase(s).
Q. What if the thieves go to the remote site and steal the remkeyservers, too?
A. The remkeyservers will use LUKS and will require a password to be enetered at boot.
As there can be multiple remkeyservers at multiple sites, it does not matter if remkeyservers have to wait for a human for occassional reboots.
Q. This is just what I need, where can I download the remkeyfs and remkeyserver code?
A. This is just a design for a solution. I haven’t written the code (yet). Email me if you’re interested in this solution.
Q. Is life easier without systemd?
A. Yes, without the systemd crypttab bug, a Devuan server’s crypttab could just curl a key from an IP-restricted remote nginx server. No new software required.
My first approach to generating random numbers lower than 1000 was to take ten random bits (which can represent 0..1023).
If that random number was less than 1000, then that was the result. If it was not, I would repeat the process until the ten bits yielded a result less than 1000. (Reusing any of those ten bits introduces bias.) This process is known as sample-and-reject.
Stef pointed out that if the result was greater than 999, then we had log2(24) bits of unused entropy.
This was an awkward amount of entropy to reuse, so I started to think if I could avoid taking it in the first place….
The algorithm below is an entropy efficient way of generating unbiased random numbers, less than a maximum value “max”, from a stream of random bits.
It is more entropy-efficient than sample-and-reject if and only if “max” has a factor which is a positive power of two.
It makes use of the fact that if the maximum has a factor which is a positive power of two, then we can decide whether a number will be too big by taking a number of random bits which are fewer than the length of the maximum (in binary). The difference in the numbers of bits is the exponent in the maximum’s power-of-two factor.
Example:
If max is 1000, it has a factor of two-to-the-power-of-three (8). 1000 is a ten bit number. But we only need to take seven bits to decide if a number 0..1023 is less than 1000.
We first generate 7 bits and shift them left three places, to give us:
r == ..._000000??_?????000
We compare this with 1000:
max == ..._00000011_11101000
So long as r >= max, we have to try again with another 7 bits.
Once r < max, we fill in the final three bits of precision:
r == ..._000000??_????????
Entropy Efficient Algorithm for Generating Random Numbers under a Maximum
constBITS_IN_U64:u32=64;fnrandom_under_max(max:u64)->u64{letmutrng=rand::thread_rng();// How many bits do we need?lettwo_power=max.trailing_zeros();letbits=BITS_IN_U64-max.leading_zeros();letsignificant_bits=bits-two_power;letmutr=u64::MAX;whiler>=max{// First collect the minimal amount of random bits which// will let us decide whether the random number is going to be less// than max.r=0;for_in0..significant_bits{// Generate one random bitletb:bool=rng.gen();// and shift it into our imprecise random number.r=r<<1|basu64;}// Shift the inexact random number into the correct position.r=r<<two_power;}// Now fill-in the precise value of this random number.r=r>>two_power;// Scrub out the insignificant zeros.for_in0..two_power{// Generate one random bitletb:bool=rng.gen();// and shift it into our random number.r=r<<1|basu64;}// Return the generated number.r}
There is no Python 2 or Python 2.7 package in Debian 12 (Bookworm).
You can successfully install it from the Debian 11 (Bullseye) packages, but it has several dependecies which are also not provided by Debian 12 (Bookworm).
Note: I have been warned that this will create a Frankendebian. It is unlikely that Debian will release clashing Debian 12 (Bookworm) versions of libffi7, libssl1.1 or the python2.7 packages. Follow these instructions at your own risk and remove the packages as soon as you have finished using python2.7 and before upgrading to Debian 13.
Any number in the small ranges maps to one 17 times larger in the large range. The smallest and largest numbers map to the smallest and largest in the large range and all are equally spaced.
In hex, the relation is trivial:
0 1 2 3 C D E F
|---|---|---|-... ...-|---|---|---|
0 11h 22h 33h CC DD EE FF
Cause
First I thought that this worked because both 16 and 256 are powers of two.
This was quickly disproved as there’s no post-and-rail way of mapping 16 onto ranges of 32, 64 or 128 values. (4,096, 65,536 and higher are possible.)
After some thought, it can be seen that any small range can neatly map onto any larger range whose size is an integer power of the size of the small range.
In CSS colours, these are 16 and 16-squared (256).
Other examples
5 and 5-squared (25):
0 1 2 3 4
|---|---|---|---|
0 6 12 18 24
3 and 3-cubed (27):
0 1 2
|---|---|
0 13 26
Size of the multiplier
If the size of the small range is S, and the larger range is S**N, then the multiplier to map between the ranges is:
N
___
\ S**n
/__
n=0
This sum corresponds exactly to repeating digits in base S.
e.g. The 3-cubed example can be written in base-3 as:
I’ve recently started a little project called ‘completion’. It’s purpose is to allow auto-completion of user input, according to a grammar. It’s readily understandable, if you already use Pyparsing. I’d like to merge this functionality into Pyparsing at some point.
Here’s the alpha code, with explanatory doctests:
#!/usr/bin/python
"""
Completion is a module for performing autocompletion of text according to a
programatically expressed grammar.
If you don't get any of that ;-) have a look at the doctests below...
>>> foo = Literal('foo')
>>> print foo.parse('f')
ParseResult(False, input=None, completions=['oo'], message=None)
>>> foo = Literal('foo')
>>> print foo.parse('ba')
ParseResult(False, input=None, completions=[], message=None)
>>> foobar = Literal('foobar')
>>> food = Literal('food')
>>> foobar_or_food = Or(foobar, food)
>>> print foobar_or_food.parse('f')
ParseResult(False, input=None, completions=['oobar', 'ood'], message=None)
>>> print foobar_or_food.parse('g')
ParseResult(False, input=None, completions=[], message=None)
>>> print foobar_or_food.parse('food')
ParseResult(True, input=, completions=[], message=None)
>>> print foobar_or_food.parse('foo')
ParseResult(False, input=None, completions=['bar', 'd'], message=None)
>>> foobar_then_food = Then(foobar, WhiteSpace(), food)
>>> print foobar_then_food.parse('fo')
ParseResult(False, input=None, completions=['obar'], message=None)
>>> print foobar_then_food.parse('')
ParseResult(False, input=None, completions=['foobar'], message=None)
>>> print foobar_then_food.parse('foobar f')
ParseResult(False, input=None, completions=['ood'], message=None)
>>> print foobar_then_food.parse('foobar')
ParseResult(False, input=, completions=[' ', '\\t'], message=None)
>>> optional_foo = Occurances(foo, min=0, max=1)
>>> optional_foo.parse("")
ParseResult(True, input=None, completions=[], message=None)
>>> optional_foo.parse("fo")
ParseResult(True, input=None, completions=[], message=None)
>>> three_or_four_foos = Occurances(foo, min=3, max=4)
>>> three_or_four_foos.parse("")
ParseResult(False, input=None, completions=['foo'], message=None)
>>> three_or_four_foos.parse("foof")
ParseResult(False, input=None, completions=['oo'], message=None)
>>> name = Alphas()
>>> name.parse("")
ParseResult(False, input=, completions=['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'], message=None)
>>> name.parse("abc")
ParseResult(True, input=, completions=['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'], message=None)
>>> bar = Literal('bar')
>>> complex = Then(name, WhiteSpace(), bar)
>>> complex.parse('f')
ParseResult(False, input=, completions=['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', ' ', '\\t'], message=None)
>>> complex = Or(Or(foo, food, foobar), Then(name, WhiteSpace(), bar))
>>> complex.parse('f')
ParseResult(False, input=None, completions=['oo', 'ood', 'oobar', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', ' ', '\\t'], message=None)
"""importsysclassParseResult:def__init__(self,success,input=None,completions=[],message=None):self.success=successself.input=inputself.completions=completionsself.message=messagedef__repr__(self):return"ParseResult(%s, input=%s, completions=%s, message=%s)"%(str(self.success),str(self.input),str(self.completions),str(self.message),)classParser:defparse(self,input):passclassLiteral(Parser):def__init__(self,text):self.text=textdefparse(self,input):ifinput.startswith(self.text):returnParseResult(True,input[len(self.text):])# if the input isn't at the beginning of self.text, then this
# cannot be autocompleted
ifnotself.text.startswith(input):returnParseResult(False)returnParseResult(False,completions=[self.text[len(input):]])classOccurances(Parser):def__init__(self,parser,min=0,max=1):self.parser=parserifmin<0:raiseException("Min must not be negative.")ifmax<min:raiseException("Max must not be less than min.")ifmax<1:raiseException("Max must be more than zero.")self.min=minself.max=maxdefparse(self,input):# see how many times parser can be applied, up to max
foriinrange(self.max):parse_result=self.parser.parse(input)input=parse_result.inputifnotparse_result.success:break#print i, self.min
ifi<self.minandself.min>0:returnparse_resultreturnParseResult(True,input)classChars(Parser):def__init__(self,chars):self.chars=charsdefparse(self,input):iflen(input)==0:returnParseResult(False,input,list(self.chars))ifinput[0]notinself.chars:returnParseResult(False,input)whilelen(input)>0andinput[0]inself.chars:input=input[1:]iflen(input)==0:returnParseResult(True,input,list(self.chars))returnParseResult(True,input)classWhiteSpace(Chars):def__init__(self):self.chars=' \t'classAlphas(Chars):def__init__(self):self.chars='abcdefghijklmnopqrstuvwxyz'classAlphanums(Chars):def__init__(self):self.chars='abcdefghijklmnopqrstuvwxyz0123456789'classThen(Parser):def__init__(self,*parsers):self.parsers=parsersdefparse(self,input):# try all the parsers in order, until one fails
completions=[]forparserinself.parsers:parse_result=parser.parse(input)completions+=parse_result.completionsifnotparse_result.success:# add the completions for successful parsings which can take more
parse_result.completions=completionsreturnparse_resultinput=parse_result.inputreturnParseResult(True,input)classOr(Parser):def__init__(self,*parsers):self.parsers=parsersdefparse(self,input):# try all the parsers and see whether any work
parse_results=map(lambdax:x.parse(input),self.parsers)# were any successful?
successful_result=Nonecompletions=[]forparse_resultinparse_results:ifparse_result.successandnotsuccessful_result:# first matched
successful_result=parse_resultcompletions+=parse_result.completionsifsuccessful_result:returnsuccessful_result# otherwise return all the completions
returnParseResult(False,None,completions)def_test():"""Test with doctest."""importdoctestdoctest.testmod()if__name__=="__main__":_test()