URL Generator (For Windows 95/98/2000/NT/ME/XP)

Based on a concept design by Plexus Digital Solutions
http://www.digital-solutions.co.uk

This documentation by Jason Sheldon, Plexus Digital Solutions.
---------------------------------------------------------------

Welcome to URL Generator for Windows 9x/2k/NT/Me/XP

URL Generator is an utility designed specifically to assist Rafabot users who need to download a large sequence of files from a server that does not reveal file listings, but where you know the file naming conventions used on the that server.

In particular, it is designed for 'directly downloadable files', where you do not want to download whole websites, or HTML files, but perhaps a range of sequential images from an online image library. Perhaps you subscribe to an image library where you have to add files to a basket, but the basket does not have a "bulk" or "download all" option - you have to  download each file individually! That could take days when you have a large library!

URL Generator will create a list of URL's based on the field values you enter. The list of URL's can then be saved from the program, and loaded into RAFABOT (General tab / Source: URLs from File). RAFABOT will then download each and every file automatically, and saves you having to manually type in each file you want to download.

Please see below the version information for tips on how to use this program effectively.

-------------------------------------------------------------------------------------

Version information:

1.0 ; 15 June 2003 ; First version.

Known Problems in current version:

* Numbers must be entered in ascending order. The program will not generate descending order URL's.

* Only numerical values can be incremented in this version. IE: You cannot increment values a-z. 

We welcome your feedback and suggestions, but please note, this is a free utility, and we will do our best to enhance it, but offer no warranties or guarantees that suggestions will be implemented.

------------------------------------------------------------------------------


The product is quite simple to use, and for best effect, it helps if you know the naming convention of the files you are trying to download. 

For example, say you know there are 3000 image files on a server, numbered sequentially, and in a variety of resolutions, like so:

http://myserver.com/catalogue/images/

img1_qtr.jpg
img2_qtr.jpg
...
...
img2999_qtr.jpg
img3000_qtr.jpg

And in higher resolutions, like so:

img1_full.jpg
img2_full.jpg
...
...
img2999_full.jpg
img3000_full.jpg

Imagine if the server does not reveal directory contents, so Rafabot would not know the files exist. You would have to manually download 6000 unique files. How Tedious!

URL Generator makes the task easy.

From the main URL Generator window, you see a number of fields where you need to build up the filename based on the common denominator in the files you want.

The server name will be part of every request, therefore, the Main URL should contain this. It would also contain the directory paths to the files you want. Based on the above example then, you would enter

http://myserver.com/catalogue/images/

into the Main URL: field.

The filename template is a group of 4 fields that are used to generate the list. The first field in the group should contain as much of the filename that is common with every URL you want to generate, starting from left to right, and up to the point where the filenames increment.

In our example above, the letters "img" of the filename are as far as we can go, because the sequential numbering portion of the filename starts from that point in the string. 

The second field, and the field directly underneath it, control the range of numbering used in the filename. 
In our example, the file range starts at image #1, up to image #3000 (img1_qtr.jpg through to img3000_qtr.jpg).

You may have already downloaded 1000 of the images, so rather than create a list of all 3000 files, you can enter "1000" in the first number field, and "3000" in the field underneath it. This will then generate a URL list of 2000 URL's, starting from 1000.

The last field in the filename template is for the extension. However, it can also be used for trailing portions of the filename, as our example shows, the image filenames might have the size of the image before the extension (_qtr and _full), so by entering everything that is common on the RIGHT side of the filename numbering, we can create a complete URL for Rafabot to use.

The "Leading Char" option is used when the filename is a fixed length, and leading characters are used in the numbering sequence.

Our first example above shows standard numbering. If they were to include leading zeros, the file names would look like this:

img0001_qtr.jpg
img0002_qtr.jpg
...
...
img2999_qtr.jpg
img3000_qtr.jpg

In order to successfully create a URL list for the above files, we would need to use the Leading Char option, by using "0" as the leading character, and "3" as the count. (Notice that the first filename has 3 leading zero's before the 1).

When you have entered the details for your list, simply click the start button. When the list is complete, you can either save it to a file, or extend the list. This is useful when say, you want to download files 1-1499, leaving 1500-2499, then starting again from 2500 to 3000. You would create the first list, with "1" and "1499" as the numerical parameters, click "Start", then before saving, change the parameters to "2500" and "3000", click "Start" again, and the URL list extends with the extra files you requested.

Another very simple example would be: say you want to generate a list of urls like
http://www.sharewareconnection.com/popular-search.php?page=1
http://www.sharewareconnection.com/popular-search.php?page=2
http://www.sharewareconnection.com/popular-search.php?page=3
..
http://www.sharewareconnection.com/popular-search.php?page=20

Enter http://www.sharewareconnection.com/popular-search.php?page= in main url box, set starting value=1 and ending value=20 , thats all! Leave all other boxes empty. Now click Start button and the list is ready.