Comments on hackerdashery: How Not to Handle Streams

I feel like the last ASP developer on the planet. ...

2008-06-23T19:51:45.779-07:00

I feel like the last ASP developer on the planet. S3 sample code and free libraries for ever platform under the sun except vbscript.

Yes there is loads for DOTNET, I have nothing against DOTNET I just don't like it for web development.

I have dozens of web based production applications in ASP. It would be nice to be able to add new S3 based features without having to migrate them dotnet.

Didn't try Perl, huh? My first transfer of a 4+ G...

2008-05-31T05:12:07.733-07:00

Didn't try Perl, huh? My first transfer of a 4+ GB file worked fine.

Zed,Thanks for pointing out http11_parser, that mi...

2008-05-16T11:13:22.301-07:00

Zed,

Thanks for pointing out http11_parser, that might be useful sometime.

I also thought it was interesting that the problem I encountered with Mongrel's API was caused by API problems further down. I look forward to seeing your C lib!

mads,That's exactly the read/write inversion I was...

2008-05-16T10:18:31.986-07:00

mads,

That's exactly the read/write inversion I was talking about. I don't want to pass in a file and have boto read from it; I want the key object to allow me to write data as I have it ready.

Did you look at send_file? As far as i can tell it...

2008-05-16T09:24:30.853-07:00

Did you look at send_file? As far as i can tell it allows you to stick it a file-like object, which it then read's from self.BufferSize bytes at a time.

mads,Looks like boto beat me to the fix. I was us...

2008-05-16T08:39:24.293-07:00

mads,

Looks like boto beat me to the fix. I was using 1.0a, which lacks that functionality. 1.2a provides the API you mentioned, as well as a more generic read() method.

That puts it ahead of AWS::S3. But, it looks like it still doesn't provide a file-like interface on the sending side. Perhaps the authors aren't aware that the Content-MD5 header is optional?

Clearly you haven't done you homework properly. fr...

2008-05-16T02:38:13.529-07:00

Clearly you haven't done you homework properly.

from boto import s3
conn = s3.connection.S3Connection('key', 'secret')
bucket = conn.get_bucket('foobar')
key = bucket.get_key('key')

for bytes in key:
other_fp.write(bytes)

have you looked at the TwistedMatrix library for P...

2008-05-16T00:08:55.831-07:00

have you looked at the TwistedMatrix library for Python. I think it has what you are looking for.

Try the lazy bytestring version of the Haskell HTT...

2008-05-15T22:56:44.185-07:00

Try the lazy bytestring version of the Haskell HTTP library:

http://www.dtek.chalmers.se/~tox/site/http.php4

It's not yet merged into the main HTTP library, and the problem with not closing is a pain to work around.

Nevertheless I've used it to build a web crawler, and to build some analysis tools for video streams (Hogg). It's pretty efficient.

Hey, Mongrel author here, the full streaming fancy...

2008-05-15T19:36:03.559-07:00

Hey, Mongrel author here, the full streaming fancy HTTP library is available for you to use just the same way mongrel does. No need to use the whole server since it seems you are just looking to do some streaming. Take a look at the heavily documented http11_parser stuff in ext.

Additionally, there's a client version of the same parser I wrote in the rfuzz.rubyforge.org project you can also use, and it handles chunked encoding.

Finally, I'm currently working on a new C lib and potential server that unifies the above parsers into a single super nice, correct, and *streamable* API.

For the record, the reason that Mongrel has to do tempfiles or StringIO is that Ruby's broke ass cgi.rb is not stream capable and insists on having the whole thing around to parse mime-types out. Otherwise, the API I wrote is fully capable of doing what you want.

Zed

Okay, I've set up Disqus. Unfortunately, they don...

2008-05-15T19:25:14.558-07:00

Okay, I've set up Disqus. Unfortunately, they don't convert old posts.

That's where the uncool kids impress me.$fp = fope...

2008-05-15T18:31:16.133-07:00

That's where the uncool kids impress me.

$fp = fopen($_REQUEST['url'], 'r', false);
// Your little bit of processing.
fpassthru($fp);

P.S. Blogger "identity" thing and CAPTCHA seriously sucks. Can't you use Disqus?