Invalid method call in credential_provider_chain to Aws::Plugins::UserAgent:Class

[ferdynator]

I am using the opensearch logstash output plugin to write data to an aws opensearch instance. Internally this gem uses the aws ruby sdk to load credentials (https://github.com/opensearch-project/logstash-output-opensearch/blob/2868d41bede5bb25ba1ef71b7c70a63ae740f445/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb#L74). I am using an assumed role web identity for authentication and thus the following code is used to load credentials from environment variables:

https://github.com/aws/aws-sdk-ruby/blame/946aefc489b6037d3fe3d680805592f178a076b9/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L220-L224

This results in:

undefined method `metric' for Aws::Plugins::UserAgent:Class

/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:175:in `assume_role_web_identity_credentials'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:13:in `block in resolve'
org/jruby/RubyArray.java:2009:in `each'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:12:in `resolve'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:87:in `aws_iam_auth_initialization'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:56:in `initialize'
org/jruby/RubyClass.java:936:in `new'

stack trace shortened.

[richardwang1124]

Hey, thanks for opening an issue. We'll be taking a look at this.

[richardwang1124]

Still investigating, it's strange that you're receiving this error since the UserAgent plugin is part of aws-sdk-core and the metric method has existed since many versions before 3.233.0. Are you able to try using another type of credentials to see if you still get the same error? Not sure how feasible this is, but if you're able to provide some code to reproduce this error that would be helpful too.

[ferdynator]

I would not expect the issue to appear on any other authentication method as this code is located explicitly inside the assume_role_web_identity_credentials method. Authentication works this way and logstash can operate normally so this is not a high-impact issue. I would assume simply metrics collection does not work in this case. I have noticed other authentication methods are using the with_metrics method instead of calling Aws::Plugins::UserAgent.metric directly but looking at the code there should not be a difference.

I have again verified that I am running the mentioned versions of the plugin (i have removed irrelevant aws sdks from the list):

$ bin/ruby -S gem list --local | grep aws
aws-sdk (3.3.0)
aws-sdk-core (3.233.0)
aws-sdk-opensearchservice (1.75.0)
aws-sigv2 (1.3.1)
aws-sigv4 (1.12.1)
logstash-integration-aws (7.2.1 java)
opensearch-aws-sigv4 (1.3.0)

[richardwang1124]

Other credential providers (such as profile assume role web id, assume role, and sso) all use the UserAgent metric method to track features, I was curious to see if you'd run into the same issue with these other providers. I tried replicating the logstash code locally but could not reproduce the error. Are you able to try a different aws-sdk-core version?

[ferdynator]

Okay I did some more investigation. There is another error happening before the mentioned one:

uninitialized constant Aws::Plugins::UserAgent

stack trace:

org/jruby/RubyModule.java:4375:in `const_missing'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:175:in `assume_role_web_identity_credentials'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:13:in `block in resolve'
org/jruby/RubyArray.java:2009:in `each'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/aws-sdk-core-3.233.0/lib/aws-sdk-core/credential_provider_chain.rb:12:in `resolve'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:87:in `aws_iam_auth_initialization'
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:56:in `initialize'

I have noticed this only happens sometimes, we have ~10 pipelines running every hour with a very similar setup and only some of them fail only sometimes with this error.

Example output pipeline configuration:

output {
    opensearch {
        hosts => ["${ES_ENDPOINT}"]
        ssl => true
        index => "${LOGSTASH_INDEX_NAME}"
        template => "<path-to-template>.json"
        template_name => "<name-of-template>"
        document_id => "%{id}"
        doc_as_upsert => true

        auth_type => {
          type => "aws_iam"
          region => '${AWS_REGION}'
        }
    }
}

I have upgraded the aws-sdk-core to 3.240 now and will check if the issue occurs again. Thank you again for taking the time to look into this 👍

[richardwang1124]

Thanks for your investigation! I think the uninitialized constant error for UserAgent could make more sense than the undefined method error, but the plugin is autoloaded as part of aws-sdk-core and should be defined...

You may have already looked into it, but did you notice anything different in the cases when pipelines would fail vs when they wouldn't?

Let me know whether upgrading core helps.

[ferdynator]

The issue has not appeared since upgrading to the newest version. I assume it was fixed. Thank you for your support and have a nice holiday season

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.