• Registered: 2007-02-07
  • Posts: 11

Topic: paperclip, S3, and multi-threading

Hello all,

I work for a company with a live site that as part of the business model saves files associated with users.  Currently those files are being saved to local disk using filecolumn.  The plan is to convert from using filecolumn to using paperclip for file storage.  Along with this conversion, the plan is to encrypt these files AND store them on S3.  Currently, the code has been written and tested with storage being on the local file system (not s3 yet).  The problem comes with the migration itself: moving, encrypting, and uploading to s3. 

Currently in production we have over 550000 files taking up 110GB worth of disk space.  Since this is a production site, a planned service interruption needs to be less than 12 hours AT MOST (in order to prevent new files from being uploaded).  Needless to say performance for this file migration is very important.  I have multi-threaded and multi-hosted this migration by using reeds.  The migration itself puts the model id's in a redis list called 'documents_in'.  From there a rake task (or hopefully several) pop an id from the list, find the document, convert it, encrypt it, and save it to s3.  I have attempted to multi-thread the rake task to improve performance.  Unfortunately it seems there is something deep down in the paperclip and aws/s3 gems that is not particularly thread safe as I regularly get errors like an http connection already being open or a close being attempted on a socket that doesn't exist.

The aws/s3 gem has an option for persistent connections; I've tried both with no success.  I have also tried mutex's around the offending code (the paperclip specific sections, see code), and while this DID prevent the errors from occurring, it also did not improve performance much since the paperclip functions are the bottle neck to begin with.  Finally, I've tried spinning these into separate processes.  This did indeed fix the problem of crashing, but since each process would spin up a full rails environment to the tune of 300MB, it only took a handful instances before the machine was brought to its knees without realizing much performance increase.

I would love someone who has experience with paperclip and the aws/s3 gem who might know where this problem is occurring to give me a hint on where to look to make this happen.  Or at the very least give me insight as to whether this is even worth pursuing at this point and if I should look to other options (like store locally and upload later, or abstract file storage and flexible access, or <insert favorite option here>).

Please find the multi-threaded rake task and a typical stack trace from the resulting failure.

Thanks in advance for your help,
Snowdall



require 'redis'

task :run_migration_workers => :environment do

  Thread.abort_on_exception = true
  threads = []

  10.times do |worker_num|
    redis = Redis.new(:thread_safe => true)
    # sleep 1

    threads << Thread.new(worker_num, redis) do |i, red|
      puts

  • Registered: 2007-02-07
  • Posts: 11

Re: paperclip, S3, and multi-threading

Seems the forum cut off my posting of code and stack trace:


require 'redis'

task :run_migration_workers => :environment do

  Thread.abort_on_exception = true
  threads = []

  10.times do |worker_num|
    redis = Redis.new(:thread_safe => true)
    # sleep 1

    threads << Thread.new(worker_num, redis) do |i, red|
      puts "thread # #{i.to_s}"

      loop {
          list, old_doc_id = red.blpop 'documents_in', 0
          puts "$ thread #{i}: popped #{old_doc_id} off redis"

          old_doc = Legacy::Document.find old_doc_id

          new_doc = Document.find(old_doc_id)

          new_doc.encrypted = true unless ["Logo", "Letterhead"].include?(new_doc.document_type)

          #mtex.lock
          new_doc.document = File.new(old_doc.file)

          puts "$ thread #{i}: uploading document #{old_doc.id}"
          new_doc.save
          #mtex.unlock

          puts "$ thread #{i}: finished uploading #{old_doc.id}"
          red.rpush 'documents_out', new_doc.id

      }

    end

  end

  threads.each {|thr| thr.join }

end




snowdall:optimis snowdall$ rake run_migration_workers
(in /Users/snowdall/Documents/Optimis/optimis)
An error occured trying to connect to MongoDB!
thread # 0thread # 1thread # 2
thread # 3
thread # 4
thread # 5
thread # 6
thread # 7
thread # 8
thread # 9


$ thread 1: popped 173 off redis$ thread 0: popped 174 off redis$ thread 2: popped 175 off redis

$ thread 4: popped 176 off redis$ thread 3: popped 177 off redis$ thread 5: popped 178 off redis
$ thread 6: popped 179 off redis

$ thread 7: popped 180 off redis
$ thread 8: popped 181 off redis

$ thread 9: popped 182 off redis

$ thread 2: uploading document 175
$ thread 0: uploading document 174
$ thread 9: uploading document 182
$ thread 5: uploading document 178
/Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/protocol.rb:178:in `write0': You have a nil object when you didn't expect it! (NoMethodError)
You might have expected an instance of Array.
The error occurred while evaluating nil.+
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/protocol.rb:153:in `write'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/protocol.rb:168:in `writing'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/protocol.rb:152:in `write'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/http.rb:1580:in `write_header'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/http.rb:1535:in `exec'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/http.rb:1049:in `__request__'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rest-client-1.5.0/lib/restclient/net_http_ext.rb:17:in `request'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/connection.rb:43:in `request'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/connection.rb:48:in `call'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/connection.rb:48:in `request'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/base.rb:69:in `request'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/base.rb:83:in `delete'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/object.rb:221:in `delete'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip/storage/s3.rb:168:in `flush_deletes'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip/storage/s3.rb:165:in `each'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip/storage/s3.rb:165:in `flush_deletes'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip/attachment.rb:147:in `save'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:363:in `send'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:363:in `save_attached_files'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:356:in `each_attachment'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:355:in `each'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:355:in `each_attachment'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:362:in `save_attached_files'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:178:in `send'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:178:in `evaluate_method'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:166:in `call'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:93:in `run'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `each'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `send'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `run'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:276:in `run_callbacks'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:344:in `callback'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:251:in `create_or_update'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/base.rb:2538:in `save_without_validation'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/validations.rb:1078:in `save_without_dirty'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/dirty.rb:79:in `save_without_transactions'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:229:in `send'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:229:in `with_transaction_returning_status'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/connection_adapters/abstract/database_statements.rb:136:in `transaction'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:182:in `transaction'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:228:in `with_transaction_returning_status'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:196:in `save'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:208:in `rollback_active_record_state!'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:196:in `save'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:48
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:34:in `loop'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:34
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:31:in `initialize'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:31:in `new'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:31
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:27:in `times'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:27
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:636:in `call'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:631:in `each'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/monitor.rb:242:in `synchronize'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2029:in `each'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/bin/rake:31
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/bin/rake:19:in `load'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/bin/rake:19
/Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/protocol.rb:70:in `rbuf_fill': stream closed (IOError)
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/timeout.rb:62:in `timeout'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/timeout.rb:93:in `timeout'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/protocol.rb:126:in `readline'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/http.rb:2026:in `read_status_line'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/http.rb:2015:in `read_new'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/net/http.rb:1051:in `__request__'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rest-client-1.5.0/lib/restclient/net_http_ext.rb:17:in `request'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/connection.rb:43:in `request'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/connection.rb:48:in `call'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/connection.rb:48:in `request'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/base.rb:69:in `request'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/base.rb:83:in `delete'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/aws/lib/aws/s3/object.rb:221:in `delete'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip/storage/s3.rb:168:in `flush_deletes'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip/storage/s3.rb:165:in `each'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip/storage/s3.rb:165:in `flush_deletes'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip/attachment.rb:147:in `save'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:363:in `send'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:363:in `save_attached_files'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:356:in `each_attachment'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:355:in `each'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:355:in `each_attachment'
    from /Users/snowdall/Documents/Optimis/optimis/vendor/plugins/paperclip/lib/paperclip.rb:362:in `save_attached_files'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:178:in `send'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:178:in `evaluate_method'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:166:in `call'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:93:in `run'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `each'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `send'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `run'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:276:in `run_callbacks'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:344:in `callback'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:251:in `create_or_update'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/base.rb:2538:in `save_without_validation'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/validations.rb:1078:in `save_without_dirty'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/dirty.rb:79:in `save_without_transactions'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:229:in `send'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:229:in `with_transaction_returning_status'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/connection_adapters/abstract/database_statements.rb:136:in `transaction'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:182:in `transaction'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:228:in `with_transaction_returning_status'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:196:in `save'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:208:in `rollback_active_record_state!'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/activerecord-2.3.5/lib/active_record/transactions.rb:196:in `save'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:48
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:34:in `loop'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:34
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:31:in `initialize'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:31:in `new'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:31
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:27:in `times'
    from /Users/snowdall/Documents/Optimis/optimis/lib/tasks/encryption_migration_worker.rake:27
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:636:in `call'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:631:in `each'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
    from /Users/snowdall/.rvm/rubies/ree-1.8.7-2010.02/lib/ruby/1.8/monitor.rb:242:in `synchronize'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2029:in `each'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/gems/rake-0.8.7/bin/rake:31
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/bin/rake:19:in `load'
    from /Users/snowdall/.rvm/gems/ree-1.8.7-2010.02@optimis/bin/rake:19

  • From: Florianópolis, SC - Brazil
  • Registered: 2008-11-07
  • Posts: 1

Re: paperclip, S3, and multi-threading

Well, you probably already figured it out or changed to another solution but here is my hints to help you with your task.

Your ruby (1.8.7 ee) is not the best version to work on a multi-thread paralel environment, you can google about it and see that because of the use of Green Threads, you will not get the performance you are thinking about.

My suggestion is that you use JRuby to that task, so you can have full thread-os mapping support and make use of all your cpu cores.

If JRuby is not possible then I suggest that at least you use ruby 1.9.3, but be warned that they have improoved their threading model but still having global locker.

Now, if you are using threads, you have to make sure that all shared resources use only acid operations.

I don't have a answer to that particular problem, but I've done some massive migrations using a thread pool  (https://github.com/brodock/work_queue ), and I can share you some knowledge about it.

First, not all Rails versions are thread-safe, so If your codebase is old, you have to upgrade it first.
Second, if you try my work_queue or any other thread solution, you are going to suffer at some time with the autoload bug. The only implementation that have fixed it is JRuby (1.7). If you want to use that version, you are going to need to install jruby-head as jruby 1.7 isn't released yet.

This autoload bug will be fixed probably on Ruby 1.9.4 or 1.9.5, as stated on this bug report http://jira.codehaus.org/browse/JRUBY-3 … tId=284322

Also note that work_queue needs a queue size otherwise you are going to suffer from memory leak (as you are going to enqueue more tasks than you queue are able to process).

I believe this information will help you get your code running.

and found out that the next problem i had to fix was related to require and dynamic