dlq 0.62 - download queue ------------------------- dlq is a simple, anonymous service that downloads file chunks whenever modem/isdn connections are idle. You retrieve files to a server on a fast pipe, using dlq to make the last (slowest) leg of the journey to a local machine; it does so in the background, without disrupting normal traffic. It is written in perl. Changes from 0.50 ----------------- - dlq now streams file and allows multiple servers - dlqadd (replacing qadd, which should be removed) no longer takes perl regular expressions, but rather the simple wildcards * and ?. - dlqrecvd now only takes a port parameter - See CHANGES file for details Usage ----- A server runs 'dlqd', which sends files located in user "spool" directories to clients. Their paths look like ~user/spool. The server will only read files in these directories. Client machines run 'dlqrecvd' which retrieves data when the local pipe is idle. The files are deposited in a shared directory, usually /usr/local/spool or somesuch. Note that no provision is made for access rights or user ownership, in the client-side spool directory. Anybody can munge files in there. 'dlqadd' adds requests to the client queue. If joe (his username too) has on server 'bub', some files ~joe/spool/afile and ~joe/spool/bfile, he can queue them for receipt (on the client) with # dlqadd bub:joe/afile bub:joe/bfile or if he likes wildcards, with # dlqadd "bub:joe/?file" or # dlqadd "bub:joe/*" The files will subsequently appear in /usr/local/spool. In fact, anybody on the client machine can grab joe's files. dlqd is an anonymous service. dlqd sends a trigger (if you engage it) whenever a file download is completed (dlq:file:). Setup ----- 1. The server side. a) Pick an unused port on your server. You might name it in /etc/services as 'dlq 9999/tcp # download queue'. b) Grab and build Dan Bernstein's tcpserver. It handles server socket logic, uid/gid and tcp access control. http://www.pobox.com/~djb/ucspi-tcp.html. c) Run tcpserver with dlqd. You probably want to pass uid/gid parameters like nobody:nogroup (it actually needs the integers themselves). # tcpserver -u65535 -g?? yourhost yourport dlqd & d) Create home spool directories as desired: ~user/spool. 2. The client side. a) Run dlqtest when the server is up and running AND THE LINE IS IDLE. The device param should look like 'ppp0' not '/dev/ppp0'. # dlqtest device serverhost serverport After a few moments, it will print the receive and transmit packet counts that are needed in the dlqrecvd config section. You might run it a few times to verify stable numbers. b) Create the common spool directory and the queue file (and assuming root runs dlqrecvd, the following permissions). # mkdir /usr/local/spool # touch /usr/local/spool/queue # chown root:users /usr/local/spool/queue # chmod u=g+rw,g=w,o= /usr/local/spool/queue c) Edit dlqrecvd's config section. i ) Set SPOOLDIR, as in (b) above. ii ) Change DEVICE appropriately (like 'ppp0' not '/dev/ppp0'). iii) Set the RECEIVE and TRANSMIT counts you got from (a) above. iv ) The default PKT_NOISE setting might be approriate for you, but check 'Idle detection' below for further details. v ) Set TRIGGERS to 1 if you have the triggers package installed, and want a trigger on file completion (dlq:file:). d) Edit dlqadd and set the SPOOLDIR and SERVERPORT variables. e) Move dlqrecvd and dlqadd to something like /usr/local/sbin and /usr/local/bin, respectively. f) Either run dlqrecvd as you wish (probably as root), # dlqrecvd serverport & or (as I do) run it in the ip-up/ip-down ppp scripts. If installed this way, dlq should properly continue interrupted transfers across line drops or reboots. in ip-up: exec /usr/local/sbin/dlqrecvd serverport in ip-down: killall dlqrecvd g) Try the queue by running 'dlqadd serverhost:user/file'. Idle detection -------------- In the absence of complete information about socket packet use, the idle detection implemented is a hack. It tries to gauge the amount of packets it uses itself, and adjust for a little bit of "background noise". A variety of factors (mru, compression, etc) will cause different packet counts, which is why 'dlqtest' is included. The PKT_NOISE value helps dlqrecvd avoid stopping at every blip on the line. For instance, my ISP might send RIP announcements every 30 seconds, so I increase noise by the two (2) packets it sends. One (1) should be your base. If you find dlqrecvd stopping too often, bump this value one or two. Security Concerns ----------------- All manner of mischievous things can be done through anonymous servers. For instance, though dlqd will only construct paths from home spool directories, users can put symbolic links to '/' and other places. This isn't a concern for me, but I might consider adding features such as "don't follow symbolic links" , or other security protections, if there is enough interest. Your best bet is to run dlqd as user/group nobody:nogroup. Credits ------- Thanks to Damien Wohl and Anders Brownworth for testing and feeback. Thanks to my family for the incessant bickering over bandwidth use that forced this solution. April 21, 1998 Jeremy Wohl / jeremyw@evantide.com http://www.evantide.com/~jeremyw/software.html