Sunday, 30 May 2010

"Slurping" data

I just came up with a better way to download all the code. Instead of using our to acquire every single file one by one, add the following to, one line, after the "echo" in the No Match section: -

wget --directory-prefix /var/www/slurped -a /var/log/squid/slurping.log -t 1 -nc -xU "XXXXXXXX" $url

Again, replace the XXXXXXXX's with your User Agent string.

Create a directory /var/www/slurped and make it writable by your proxy user.

Now, every time your device accesses a file from panasonic (or anywhere else for that matter), it will be pulled down automatically into this holding area. Any files you want to modify, simply move it to it's equivalent directory in /var/www. e.g. cp /var/www/slurped/ /var/www/ (after mkdir obviously :D )

NOTE: This will slow down your experience, since it's downloading everything twice! Also, it doesn't pull down everything (some URL's are too long - but that's handy because it would be pulling down full mpeg files!). Just keep an eye on /var/log/squid/slurping.log for any errors.

No comments:

Post a Comment