Downloader is a service providing asynchronous and rate-limited download
capabilities. It is entirely written in Go and backed by
Redis as a metadata storage backend.
Visit the wiki for documentation.
Getting Started
Make sure that you have a working Go
environment and that you have configured your
$GOPATH
correctly.
Clone the repository or go get
it with:
$ go get github.com/skroutz/downloader
Dependencies are managed using dep. So after
cloning the project, just run make
to install the dependencies and build the
package from the project directory $GOPATH/src/github.com/skroutz/downloader/
:
$ make
Finally, to install the Downloader, just run:
$ make install
For more information about the supported make targets, please read the
Synopsis sections in the package's Makefile.
Enjoy! :)
API
Endpoints
POST /download
Enqueue a new download.
Expects JSON encoded params.
Parameters:
aggr_id
: string, Grouping identifier for the download job.
aggr_limit
: int, Max concurrency limit for the specified group ( aggr_id ).
aggr_proxy
: ( optional ) string, HTTP proxy configuration. It is set up on aggregation level and it cannot be updated for an existing aggregation.
url
: string, The URL pointing to the resource that will get downloaded.
callback_url
: string, The endpoint on which the job callback request will be performed.
extra
: ( optional ) string, Client provided metadata that get passed back in the callback.
mime_type
: ( optional ) string, series of mime types that the download is going to be verified against.
download_timeout
: ( optional ) int, HTTP client timeout per Job, in seconds.
request_headers
: ( optional ) object{string => string}, HTTP Request Headers per job.
Output: JSON document containing the download's id e.g, {"id":"NSb4FOAs9fVaQw"}
GET /hb
Acts as a heartbeat for the downloader instance.
Depending on the existence of a certain file on disk returns HTTP status code 503 if path exists, 200 otherwise.
POST /retry/:job_id
Retries the callback of the job with the specified id.
Returns HTTP status 201 on success.
GET /dashboard/aggregations
Returns a JSON list of aggregations with pending jobs.
Output: JSON array of aggregation names and their pending jobs [{"name":"jobs:super-aggregation","size":17}]
Usage
Configuration
A sample configuration can be found in config.json
file.
It is important to note that the Notifier component depends on the config file
in order to correctly enable the backends that are defined in the config's backends
key.
If no backends are given the Notifier will throw an error and exit with a non-zero code.
If you want to enable the http backend add the http
key along with its timeout
value.
If you want to enable the kafka backend add the kafka
key along with your desired configuration.
Below you can find examples of jobs enqueueing and callbacks payloads
Example using http
as backend
$ curl -d '{"aggr_id":"aggrFooBar", "aggr_limit":8, "url":"https://httpbin.org/image/png", "callback_type": "http", "callback_dst":"https://callback.example.com", "extra":"foobar", "mime_type": "!image/vnd.adobe.photoshop,image/*", "request_headers": {"Accept":"image/png,image/jpeg,image/*,*/*","User-Agent":"Downloader-Agent"}}' https://downloader.example.com/download
# => {"id":"NSb4FOAs9fVaQw"}
Example using kafka
as backend
Suppose you have already configured a kafka cluster and created a topic dwl_images
.
$ curl -d '{"aggr_id":"aggrFooBar", "aggr_limit":8, "url":"http://httpbin.org/image/png", "callback_type":"kafka" ,"callback_dst":"dwl_images", "extra":"foobar", "mime_type": "!image/vnd.adobe.photoshop,image/*", "request_headers": {"Accept":"image/png,image/jpeg,image/*,*/*","User-Agent":"Downloader-Agent"}}' http://localhost:8000/download
# => {"id":"Hl2VErjyL5UK9A"}
Example Callback payloads:
- Successful download callback:
{
"success":true,
"error":"",
"extra":"foobar",
"resource_url":"https://httpbin.org/image/png",
"download_url":"http://localhost/foo/6QE/6QEywYsd0jrKAg",
"job_id":"6QEywYsd0jrKAg",
"response_code":200,
"delivered":true,
"delivery_error":""
}
Unsuccessful Callback Examples:
{
"success":false,
"error":"Received Status code 404",
"extra":"foobar",
"resource_url":"https://httpbin.org/image/png",
"download_url":"http://localhost/foo/6QE/6QEywYsd0jrKAg",
"job_id":"6QEywYsd0jrKAg",
"response_code":404,
"delivered":true,
"delivery_error":""
}
{
"success":false,
"error":"TLS Error Occured: dial: x509: certificate signed by unknown authority",
"extra":"foobar",
"resource_url":"https://httpbin.org/image/png",
"download_url":"http://localhost/foo/6QE/6QEywYsd0jrKAg",
"job_id":"6QEywYsd0jrKAg",
"response_code":0,
"delivered":true,
"delivery_error":""
}
{
"success":false,
"error":"Expected mime-type to be (image/jpeg), found (image/png)",
"extra":"foobar",
"resource_url":"https://httpbin.org/image/png",
"download_url":"http://localhost/foo/6QE/6QEywYsd0jrKAg",
"job_id":"6QEywYsd0jrKAg",
"response_code":200,
"delivered":true,
"delivery_error":""
}
For http as a notifier backend any 2XX response to the callback POST marks the callback as successful for the current job.
For kafka as a notifier backend, we monitor kafka's Events
channel and mark a job's callback as successful if the delivery report
of a job's callback has been received and has no errors.
Web UI
An informational Web UI is served on the default route of the API.
Displayed Info:
- Downloader instances currently running.
- Current active aggregations.
- General statistics reported by the downloader.
Development
⚠ You should have a running Redis instance in order to be able to
run the downloader's tests. Make sure to update the corresponding setting in
the configuration file.
In case you haven't done it already (as described in the Getting
Started section), run:
$ make
to manage the project's dependencies. Now, you will have a fully functioning
development environment.
To run the tests and perform various package-related checks, just run:
$ make check
Credits
downloader is released under the GNU General Public License version 3. See COPYING.