|
»
»
»
Duplicate message delivered to consumers on queue with kaha persistence
|
View:
New views
17 Messages
—
Rating Filter:
Alert me
|
 |
Duplicate message delivered to consumers on queue with kaha persistence
by David Sitsky-2
::
Rate this Message:
I have discovered an issue with the latest activemq 5.0 (revision
581510) which seems to have been present for some time, at least for the last month. I have an application using transactions, and kaha persistence, and two consumers feeding off a queue. I sometimes see the same message being sent to two consumers. All transactions were committed (no rollbacks), and as you can see by the debug dump, the message has the same message-ID, but was delivered to two separate consumers, running on separate threads, using sync receives with a prefetch of 0, with separate sessions of course. Is this a known problem? It doesn't always happen - seems to be a race condition. If I turn off persistence, I don't seem to see it. Here is the message being delivered to consumer 1: 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer - ID:tamarama-62436-1191396093006-2:2:3:2 received message: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination = queue://worker-items, message = ActiveMQObjectMessage {commandId = 50, responseRequired = false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = null, originalTransactionId = null, producerId = ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue://worker-items, transactionId = TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp = 1191396097286, arrival = 0, brokerInTime = 1191396097290, brokerOutTime = 1191396097303, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0} Here is the same message (check the message ID) being delivered to consumer 2: 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer - ID:tamarama-62436-1191396093006-2:2:1:2 received message: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination = queue://worker-items, message = ActiveMQObjectMessage {commandId = 50, responseRequired = false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = null, originalTransactionId = null, producerId = ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue://worker-items, transactionId = TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp = 1191396097286, arrival = 0, brokerInTime = 1191396097290, brokerOutTime = 1191396097417, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@1a881cc, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0} I'll need to do more investigations tomorrow, but if anybody has any ideas, I'd appreciate some insights. -- Cheers, David Nuix Pty Ltd Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699 Web: http://www.nuix.com Fax: +61 2 9212 6902 |
|
|
Re: Duplicate message delivered to consumers on queue with kaha persistence
by rajdavies
::
Rate this Message:
Do you get the same using AMQ store (the default in 5.0) - its the
prefered store over kaha in ActiveMQ 5.0 cheers, Rob http://rajdavies.blogspot.com/ On Oct 3, 2007, at 8:50 AM, David Sitsky wrote: > I have discovered an issue with the latest activemq 5.0 (revision > 581510) which seems to have been present for some time, at least > for the last month. > > I have an application using transactions, and kaha persistence, and > two consumers feeding off a queue. > > I sometimes see the same message being sent to two consumers. All > transactions were committed (no rollbacks), and as you can see by > the debug dump, the message has the same message-ID, but was > delivered to two separate consumers, running on separate threads, > using sync receives with a prefetch of 0, with separate sessions of > course. > > Is this a known problem? It doesn't always happen - seems to be a > race condition. If I turn off persistence, I don't seem to see it. > > Here is the message being delivered to consumer 1: > > 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd] > 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer - > ID:tamarama-62436-1191396093006-2:2:3:2 received message: > MessageDispatch {commandId = 0, responseRequired = false, > consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination = > queue://worker-items, message = ActiveMQObjectMessage {commandId = > 50, responseRequired = false, messageId = > ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = > null, originalTransactionId = null, producerId = > ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue:// > worker-items, transactionId = > TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp > = 1191396097286, arrival = 0, brokerInTime = 1191396097290, > brokerOutTime = 1191396097303, correlationId = null, replyTo = > null, persistent = true, type = null, priority = 4, groupID = null, > groupSequence = 0, targetConsumerId = null, compressed = false, > userID = null, content = > org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties > = null, dataStructure = null, redeliveryCounter = 0, size = 0, > properties = null, readOnlyProperties = true, readOnlyBody = true, > droppable = false}, redeliveryCounter = 0} > > Here is the same message (check the message ID) being delivered to > consumer 2: > > 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1] > 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer - > ID:tamarama-62436-1191396093006-2:2:1:2 received message: > MessageDispatch {commandId = 0, responseRequired = false, > consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination = > queue://worker-items, message = ActiveMQObjectMessage {commandId = > 50, responseRequired = false, messageId = > ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = > null, originalTransactionId = null, producerId = > ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue:// > worker-items, transactionId = > TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp > = 1191396097286, arrival = 0, brokerInTime = 1191396097290, > brokerOutTime = 1191396097417, correlationId = null, replyTo = > null, persistent = true, type = null, priority = 4, groupID = null, > groupSequence = 0, targetConsumerId = null, compressed = false, > userID = null, content = > org.apache.activemq.util.ByteSequence@1a881cc, marshalledProperties > = null, dataStructure = null, redeliveryCounter = 0, size = 0, > properties = null, readOnlyProperties = true, readOnlyBody = true, > droppable = false}, redeliveryCounter = 0} > > I'll need to do more investigations tomorrow, but if anybody has > any ideas, I'd appreciate some insights. > > -- > Cheers, > David > > Nuix Pty Ltd > Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 > 0699 > Web: http://www.nuix.com Fax: +61 2 9212 > 6902 |
|
|
Re: Duplicate message delivered to consumers on queue with kaha persistence
by David Sitsky-2
::
Rate this Message:
Hi Rob,
Looks like the AMQ store works correctly - thanks for the advice. Cheers, David Rob Davies wrote: > Do you get the same using AMQ store (the default in 5.0) - its the > prefered store over kaha in ActiveMQ 5.0 > > > > cheers, > > Rob > > http://rajdavies.blogspot.com/ > > > > On Oct 3, 2007, at 8:50 AM, David Sitsky wrote: > >> I have discovered an issue with the latest activemq 5.0 (revision >> 581510) which seems to have been present for some time, at least for >> the last month. >> >> I have an application using transactions, and kaha persistence, and >> two consumers feeding off a queue. >> >> I sometimes see the same message being sent to two consumers. All >> transactions were committed (no rollbacks), and as you can see by the >> debug dump, the message has the same message-ID, but was delivered to >> two separate consumers, running on separate threads, using sync >> receives with a prefetch of 0, with separate sessions of course. >> >> Is this a known problem? It doesn't always happen - seems to be a >> race condition. If I turn off persistence, I don't seem to see it. >> >> Here is the message being delivered to consumer 1: >> >> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd] >> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >> ID:tamarama-62436-1191396093006-2:2:3:2 received message: >> MessageDispatch {commandId = 0, responseRequired = false, consumerId = >> ID:tamarama-62436-1191396093006-2:2:3:2, destination = >> queue://worker-items, message = ActiveMQObjectMessage {commandId = 50, >> responseRequired = false, messageId = >> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = null, >> originalTransactionId = null, producerId = >> ID:tamarama-62436-1191396093006-2:2:1:3, destination = >> queue://worker-items, transactionId = >> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp = >> 1191396097286, arrival = 0, brokerInTime = 1191396097290, >> brokerOutTime = 1191396097303, correlationId = null, replyTo = null, >> persistent = true, type = null, priority = 4, groupID = null, >> groupSequence = 0, targetConsumerId = null, compressed = false, userID >> = null, content = org.apache.activemq.util.ByteSequence@fa8cbc, >> marshalledProperties = null, dataStructure = null, redeliveryCounter = >> 0, size = 0, properties = null, readOnlyProperties = true, >> readOnlyBody = true, droppable = false}, redeliveryCounter = 0} >> >> Here is the same message (check the message ID) being delivered to >> consumer 2: >> >> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1] >> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >> ID:tamarama-62436-1191396093006-2:2:1:2 received message: >> MessageDispatch {commandId = 0, responseRequired = false, consumerId = >> ID:tamarama-62436-1191396093006-2:2:1:2, destination = >> queue://worker-items, message = ActiveMQObjectMessage {commandId = 50, >> responseRequired = false, messageId = >> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = null, >> originalTransactionId = null, producerId = >> ID:tamarama-62436-1191396093006-2:2:1:3, destination = >> queue://worker-items, transactionId = >> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp = >> 1191396097286, arrival = 0, brokerInTime = 1191396097290, >> brokerOutTime = 1191396097417, correlationId = null, replyTo = null, >> persistent = true, type = null, priority = 4, groupID = null, >> groupSequence = 0, targetConsumerId = null, compressed = false, userID >> = null, content = org.apache.activemq.util.ByteSequence@1a881cc, >> marshalledProperties = null, dataStructure = null, redeliveryCounter = >> 0, size = 0, properties = null, readOnlyProperties = true, >> readOnlyBody = true, droppable = false}, redeliveryCounter = 0} >> >> I'll need to do more investigations tomorrow, but if anybody has any >> ideas, I'd appreciate some insights. >> >> -- >> Cheers, >> David >> >> Nuix Pty Ltd >> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699 >> Web: http://www.nuix.com Fax: +61 2 9212 6902 -- Cheers, David Nuix Pty Ltd Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699 Web: http://www.nuix.com Fax: +61 2 9212 6902 |
|
|
Re: Duplicate message delivered to consumers on queue with kaha persistence
by PieterN
::
Rate this Message:
This might be related to issue AMQ-1389, where I also see duplicate messages when using the Kaha persistence adapter, although in a bit different scenario. For me, the AMQ store works correctly as well, but I still believe this is a bug in the Kaha adapter. Regards, Pieter |
|
|
Re: Duplicate message delivered to consumers on queue with kaha persistence
by PieterN
::
Rate this Message:
Sorry I just see this issue is resolved -- haven't tried it yet though. Pieter |
|
|
Re: Duplicate message delivered to consumers on queue with kaha persistence
by Daryl Richter-3
::
Rate this Message:
On Oct 3, 2007, at 7:30 AM, Rob Davies wrote:
> Do you get the same using AMQ store (the default in 5.0) - its the > prefered store over kaha in ActiveMQ 5.0 Hmm. When I start a very recent (revision 581630) build of ActiveMQ with completely default config I get (among others) the following lines in my log: INFO KahaStore - Kaha Store using data directory /usr/local/apache/activemq-5.0-581630/data/localhost/kr- store/state INFO AMQPersistenceAdapter - Active data files: [] WARN AMQPersistenceAdapter - The ReferenceStore is not valid - recovering ... INFO KahaStore - Kaha Store successfully deleted data directory /usr/local/apache/activemq-5.0-581630/data/ localhost/kr-store/data INFO AMQPersistenceAdapter - Journal Recovery Started from: DataManager:(data-) INFO AMQPersistenceAdapter - Recovered 0 operations from redo log in 0.0080 seconds. INFO AMQPersistenceAdapter - Finished recovering the ReferenceStore INFO KahaStore - Kaha Store using data directory /usr/local/apache/activemq-5.0-581630/data/localhost/kr- store/data So, which do I have? AMQStore or Kaha? > > > > cheers, > > Rob > > http://rajdavies.blogspot.com/ > > > > On Oct 3, 2007, at 8:50 AM, David Sitsky wrote: > >> I have discovered an issue with the latest activemq 5.0 (revision >> 581510) which seems to have been present for some time, at least >> for the last month. >> >> I have an application using transactions, and kaha persistence, >> and two consumers feeding off a queue. >> >> I sometimes see the same message being sent to two consumers. All >> transactions were committed (no rollbacks), and as you can see by >> the debug dump, the message has the same message-ID, but was >> delivered to two separate consumers, running on separate threads, >> using sync receives with a prefetch of 0, with separate sessions >> of course. >> >> Is this a known problem? It doesn't always happen - seems to be a >> race condition. If I turn off persistence, I don't seem to see it. >> >> Here is the message being delivered to consumer 1: >> >> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd] >> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >> ID:tamarama-62436-1191396093006-2:2:3:2 received message: >> MessageDispatch {commandId = 0, responseRequired = false, >> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination >> = queue://worker-items, message = ActiveMQObjectMessage {commandId >> = 50, responseRequired = false, messageId = >> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = >> null, originalTransactionId = null, producerId = >> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue:// >> worker-items, transactionId = >> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, >> timestamp = 1191396097286, arrival = 0, brokerInTime = >> 1191396097290, brokerOutTime = 1191396097303, correlationId = >> null, replyTo = null, persistent = true, type = null, priority = >> 4, groupID = null, groupSequence = 0, targetConsumerId = null, >> compressed = false, userID = null, content = >> org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties >> = null, dataStructure = null, redeliveryCounter = 0, size = 0, >> properties = null, readOnlyProperties = true, readOnlyBody = true, >> droppable = false}, redeliveryCounter = 0} >> >> Here is the same message (check the message ID) being delivered to >> consumer 2: >> >> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1] >> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >> ID:tamarama-62436-1191396093006-2:2:1:2 received message: >> MessageDispatch {commandId = 0, responseRequired = false, >> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination >> = queue://worker-items, message = ActiveMQObjectMessage {commandId >> = 50, responseRequired = false, messageId = >> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = >> null, originalTransactionId = null, producerId = >> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue:// >> worker-items, transactionId = >> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, >> timestamp = 1191396097286, arrival = 0, brokerInTime = >> 1191396097290, brokerOutTime = 1191396097417, correlationId = >> null, replyTo = null, persistent = true, type = null, priority = >> 4, groupID = null, groupSequence = 0, targetConsumerId = null, >> compressed = false, userID = null, content = >> org.apache.activemq.util.ByteSequence@1a881cc, >> marshalledProperties = null, dataStructure = null, >> redeliveryCounter = 0, size = 0, properties = null, >> readOnlyProperties = true, readOnlyBody = true, droppable = >> false}, redeliveryCounter = 0} >> >> I'll need to do more investigations tomorrow, but if anybody has >> any ideas, I'd appreciate some insights. >> >> -- >> Cheers, >> David >> >> Nuix Pty Ltd >> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 >> 9280 0699 >> Web: http://www.nuix.com Fax: +61 2 >> 9212 6902 > -- Daryl http://itsallsemantics.com "I see this as coming down to mutual respect. I want to respect the others I communicate with enough to tell them my truth without reservation and I want to respect them enough to listen to their truth. I want to respect their good intentions enough to believe that we can work past our disagreements." -- Kent Beck, 2006 |
|
|
Re: Duplicate message delivered to consumers on queue with kaha persistence
by rajdavies
::
Rate this Message:
AMQ Store is the combination of the journal (which itself now uses
Kaha) and Kaha to store message references cheers, Rob http://rajdavies.blogspot.com/ On Oct 4, 2007, at 8:04 PM, Daryl Richter wrote: > On Oct 3, 2007, at 7:30 AM, Rob Davies wrote: > >> Do you get the same using AMQ store (the default in 5.0) - its the >> prefered store over kaha in ActiveMQ 5.0 > > > Hmm. When I start a very recent (revision 581630) build of > ActiveMQ with completely default config I get (among others) the > following lines in my log: > > INFO KahaStore - Kaha Store using data > directory /usr/local/apache/activemq-5.0-581630/data/localhost/kr- > store/state > INFO AMQPersistenceAdapter - Active data files: [] > WARN AMQPersistenceAdapter - The ReferenceStore is not > valid - recovering ... > INFO KahaStore - Kaha Store successfully > deleted data directory /usr/local/apache/activemq-5.0-581630/data/ > localhost/kr-store/data > INFO AMQPersistenceAdapter - Journal Recovery Started > from: DataManager:(data-) > INFO AMQPersistenceAdapter - Recovered 0 operations from > redo log in 0.0080 seconds. > INFO AMQPersistenceAdapter - Finished recovering the > ReferenceStore > INFO KahaStore - Kaha Store using data > directory /usr/local/apache/activemq-5.0-581630/data/localhost/kr- > store/data > > So, which do I have? AMQStore or Kaha? > > >> >> >> >> cheers, >> >> Rob >> >> http://rajdavies.blogspot.com/ >> >> >> >> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote: >> >>> I have discovered an issue with the latest activemq 5.0 (revision >>> 581510) which seems to have been present for some time, at least >>> for the last month. >>> >>> I have an application using transactions, and kaha persistence, >>> and two consumers feeding off a queue. >>> >>> I sometimes see the same message being sent to two consumers. >>> All transactions were committed (no rollbacks), and as you can >>> see by the debug dump, the message has the same message-ID, but >>> was delivered to two separate consumers, running on separate >>> threads, using sync receives with a prefetch of 0, with separate >>> sessions of course. >>> >>> Is this a known problem? It doesn't always happen - seems to be >>> a race condition. If I turn off persistence, I don't seem to see >>> it. >>> >>> Here is the message being delivered to consumer 1: >>> >>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd] >>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >>> ID:tamarama-62436-1191396093006-2:2:3:2 received message: >>> MessageDispatch {commandId = 0, responseRequired = false, >>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination >>> = queue://worker-items, message = ActiveMQObjectMessage >>> {commandId = 50, responseRequired = false, messageId = >>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = >>> null, originalTransactionId = null, producerId = >>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue:// >>> worker-items, transactionId = >>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, >>> timestamp = 1191396097286, arrival = 0, brokerInTime = >>> 1191396097290, brokerOutTime = 1191396097303, correlationId = >>> null, replyTo = null, persistent = true, type = null, priority = >>> 4, groupID = null, groupSequence = 0, targetConsumerId = null, >>> compressed = false, userID = null, content = >>> org.apache.activemq.util.ByteSequence@fa8cbc, >>> marshalledProperties = null, dataStructure = null, >>> redeliveryCounter = 0, size = 0, properties = null, >>> readOnlyProperties = true, readOnlyBody = true, droppable = >>> false}, redeliveryCounter = 0} >>> >>> Here is the same message (check the message ID) being delivered >>> to consumer 2: >>> >>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1] >>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >>> ID:tamarama-62436-1191396093006-2:2:1:2 received message: >>> MessageDispatch {commandId = 0, responseRequired = false, >>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination >>> = queue://worker-items, message = ActiveMQObjectMessage >>> {commandId = 50, responseRequired = false, messageId = >>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = >>> null, originalTransactionId = null, producerId = >>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = queue:// >>> worker-items, transactionId = >>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, >>> timestamp = 1191396097286, arrival = 0, brokerInTime = >>> 1191396097290, brokerOutTime = 1191396097417, correlationId = >>> null, replyTo = null, persistent = true, type = null, priority = >>> 4, groupID = null, groupSequence = 0, targetConsumerId = null, >>> compressed = false, userID = null, content = >>> org.apache.activemq.util.ByteSequence@1a881cc, >>> marshalledProperties = null, dataStructure = null, >>> redeliveryCounter = 0, size = 0, properties = null, >>> readOnlyProperties = true, readOnlyBody = true, droppable = >>> false}, redeliveryCounter = 0} >>> >>> I'll need to do more investigations tomorrow, but if anybody has >>> any ideas, I'd appreciate some insights. >>> >>> -- >>> Cheers, >>> David >>> >>> Nuix Pty Ltd >>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 >>> 9280 0699 >>> Web: http://www.nuix.com Fax: +61 2 >>> 9212 6902 >> > > -- > Daryl > http://itsallsemantics.com > > "I see this as coming down to mutual respect. I want to respect the > others I communicate with enough to tell them my truth without > reservation and I want to respect them enough to listen to their > truth. I want to respect their good intentions enough to believe > that we can work past our disagreements." > > -- Kent Beck, 2006 > |
|
|
RecoveryListenerAdapter - Message id ID... could not be recovered from the data store: WAS: [Duplicate message delivered to consumers on queue with kaha persistence]
by David Sitsky-2
::
Rate this Message:
Hi Rob,
I think I spoke too soon. While using AMQ store fixed the initial issue I was dealing with, it seems to have other issues in my application, where I see the following message before everything hangs: RecoveryListenerAdapter - Message id ID... could not be recovered from the data store! I created a JIRA record which contains the detailed debug message logs: https://issues.apache.org/activemq/browse/AMQ-1445 FWIW, JDBC persistence works fine. Any ideas what might be wrong? Cheers, David David Sitsky wrote: > Hi Rob, > > Looks like the AMQ store works correctly - thanks for the advice. > > Cheers, > David > > Rob Davies wrote: >> Do you get the same using AMQ store (the default in 5.0) - its the >> prefered store over kaha in ActiveMQ 5.0 >> >> >> >> cheers, >> >> Rob >> >> http://rajdavies.blogspot.com/ >> >> >> >> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote: >> >>> I have discovered an issue with the latest activemq 5.0 (revision >>> 581510) which seems to have been present for some time, at least for >>> the last month. >>> >>> I have an application using transactions, and kaha persistence, and >>> two consumers feeding off a queue. >>> >>> I sometimes see the same message being sent to two consumers. All >>> transactions were committed (no rollbacks), and as you can see by the >>> debug dump, the message has the same message-ID, but was delivered to >>> two separate consumers, running on separate threads, using sync >>> receives with a prefetch of 0, with separate sessions of course. >>> >>> Is this a known problem? It doesn't always happen - seems to be a >>> race condition. If I turn off persistence, I don't seem to see it. >>> >>> Here is the message being delivered to consumer 1: >>> >>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd] >>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >>> ID:tamarama-62436-1191396093006-2:2:3:2 received message: >>> MessageDispatch {commandId = 0, responseRequired = false, consumerId >>> = ID:tamarama-62436-1191396093006-2:2:3:2, destination = >>> queue://worker-items, message = ActiveMQObjectMessage {commandId = >>> 50, responseRequired = false, messageId = >>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = >>> null, originalTransactionId = null, producerId = >>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = >>> queue://worker-items, transactionId = >>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp = >>> 1191396097286, arrival = 0, brokerInTime = 1191396097290, >>> brokerOutTime = 1191396097303, correlationId = null, replyTo = null, >>> persistent = true, type = null, priority = 4, groupID = null, >>> groupSequence = 0, targetConsumerId = null, compressed = false, >>> userID = null, content = >>> org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties = >>> null, dataStructure = null, redeliveryCounter = 0, size = 0, >>> properties = null, readOnlyProperties = true, readOnlyBody = true, >>> droppable = false}, redeliveryCounter = 0} >>> >>> Here is the same message (check the message ID) being delivered to >>> consumer 2: >>> >>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1] >>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >>> ID:tamarama-62436-1191396093006-2:2:1:2 received message: >>> MessageDispatch {commandId = 0, responseRequired = false, consumerId >>> = ID:tamarama-62436-1191396093006-2:2:1:2, destination = >>> queue://worker-items, message = ActiveMQObjectMessage {commandId = >>> 50, responseRequired = false, messageId = >>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = >>> null, originalTransactionId = null, producerId = >>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = >>> queue://worker-items, transactionId = >>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp = >>> 1191396097286, arrival = 0, brokerInTime = 1191396097290, >>> brokerOutTime = 1191396097417, correlationId = null, replyTo = null, >>> persistent = true, type = null, priority = 4, groupID = null, >>> groupSequence = 0, targetConsumerId = null, compressed = false, >>> userID = null, content = >>> org.apache.activemq.util.ByteSequence@1a881cc, marshalledProperties = >>> null, dataStructure = null, redeliveryCounter = 0, size = 0, >>> properties = null, readOnlyProperties = true, readOnlyBody = true, >>> droppable = false}, redeliveryCounter = 0} >>> >>> I'll need to do more investigations tomorrow, but if anybody has any >>> ideas, I'd appreciate some insights. >>> >>> -- >>> Cheers, >>> David >>> >>> Nuix Pty Ltd >>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699 >>> Web: http://www.nuix.com Fax: +61 2 9212 6902 > > -- Cheers, David Nuix Pty Ltd Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699 Web: http://www.nuix.com Fax: +61 2 9212 6902 |
|
|
Re: RecoveryListenerAdapter - Message id ID... could not be recovered from the data store: WAS: [Duplicate message delivered to consumers on queue with kaha persistence]
by rajdavies
::
Rate this Message:
Hi David,
I've had a look through the log you've attached to AMQ-1445 - but I think i'll need some more information to successfully resolve whats going on. Obviously there's an attempt to retrieve a message that's already been deleted - I just need to find out why ;) Could I trouble you to try and devise a junit test case to reproduce? thanks, Rob On Oct 5, 2007, at 7:37 AM, David Sitsky wrote: > Hi Rob, > > I think I spoke too soon. While using AMQ store fixed the initial > issue I was dealing with, it seems to have other issues in my > application, where I see the following message before everything > hangs: > > RecoveryListenerAdapter - Message id ID... could not be recovered > from the data store! > > I created a JIRA record which contains the detailed debug message > logs: > > https://issues.apache.org/activemq/browse/AMQ-1445 > > FWIW, JDBC persistence works fine. Any ideas what might be wrong? > > Cheers, > David > > David Sitsky wrote: >> Hi Rob, >> Looks like the AMQ store works correctly - thanks for the advice. >> Cheers, >> David >> Rob Davies wrote: >>> Do you get the same using AMQ store (the default in 5.0) - its >>> the prefered store over kaha in ActiveMQ 5.0 >>> >>> >>> >>> cheers, >>> >>> Rob >>> >>> http://rajdavies.blogspot.com/ >>> >>> >>> >>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote: >>> >>>> I have discovered an issue with the latest activemq 5.0 >>>> (revision 581510) which seems to have been present for some >>>> time, at least for the last month. >>>> >>>> I have an application using transactions, and kaha persistence, >>>> and two consumers feeding off a queue. >>>> >>>> I sometimes see the same message being sent to two consumers. >>>> All transactions were committed (no rollbacks), and as you can >>>> see by the debug dump, the message has the same message-ID, but >>>> was delivered to two separate consumers, running on separate >>>> threads, using sync receives with a prefetch of 0, with separate >>>> sessions of course. >>>> >>>> Is this a known problem? It doesn't always happen - seems to be >>>> a race condition. If I turn off persistence, I don't seem to >>>> see it. >>>> >>>> Here is the message being delivered to consumer 1: >>>> >>>> 2007-10-03 17:21:37,310 [Worker >>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG >>>> org.apache.activemq.ActiveMQMessageConsumer - >>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message: >>>> MessageDispatch {commandId = 0, responseRequired = false, >>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, >>>> destination = queue://worker-items, message = >>>> ActiveMQObjectMessage {commandId = 50, responseRequired = false, >>>> messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1, >>>> originalDestination = null, originalTransactionId = null, >>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3, >>>> destination = queue://worker-items, transactionId = >>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, >>>> timestamp = 1191396097286, arrival = 0, brokerInTime = >>>> 1191396097290, brokerOutTime = 1191396097303, correlationId = >>>> null, replyTo = null, persistent = true, type = null, priority = >>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null, >>>> compressed = false, userID = null, content = >>>> org.apache.activemq.util.ByteSequence@fa8cbc, >>>> marshalledProperties = null, dataStructure = null, >>>> redeliveryCounter = 0, size = 0, properties = null, >>>> readOnlyProperties = true, readOnlyBody = true, droppable = >>>> false}, redeliveryCounter = 0} >>>> >>>> Here is the same message (check the message ID) being delivered >>>> to consumer 2: >>>> >>>> 2007-10-03 17:21:37,422 [Worker >>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG >>>> org.apache.activemq.ActiveMQMessageConsumer - >>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message: >>>> MessageDispatch {commandId = 0, responseRequired = false, >>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, >>>> destination = queue://worker-items, message = >>>> ActiveMQObjectMessage {commandId = 50, responseRequired = false, >>>> messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1, >>>> originalDestination = null, originalTransactionId = null, >>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3, >>>> destination = queue://worker-items, transactionId = >>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, >>>> timestamp = 1191396097286, arrival = 0, brokerInTime = >>>> 1191396097290, brokerOutTime = 1191396097417, correlationId = >>>> null, replyTo = null, persistent = true, type = null, priority = >>>> 4, groupID = null, groupSequence = 0, targetConsumerId = null, >>>> compressed = false, userID = null, content = >>>> org.apache.activemq.util.ByteSequence@1a881cc, >>>> marshalledProperties = null, dataStructure = null, >>>> redeliveryCounter = 0, size = 0, properties = null, >>>> readOnlyProperties = true, readOnlyBody = true, droppable = >>>> false}, redeliveryCounter = 0} >>>> >>>> I'll need to do more investigations tomorrow, but if anybody has >>>> any ideas, I'd appreciate some insights. >>>> >>>> -- >>>> Cheers, >>>> David >>>> >>>> Nuix Pty Ltd >>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 >>>> 9280 0699 >>>> Web: http://www.nuix.com Fax: +61 2 >>>> 9212 6902 > > > -- > Cheers, > David > > Nuix Pty Ltd > Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 > 0699 > Web: http://www.nuix.com Fax: +61 2 9212 > 6902 |
|
|
Re: RecoveryListenerAdapter - Message id ID... could not be recovered from the data store: WAS: [Duplicate message delivered to consumers on queue with kaha persistence]
by David Sitsky-2
::
Rate this Message:
Hi Rob,
Looks like you have fixed this issue - many thanks. Cheers, David Rob Davies wrote: > Hi David, > > I've had a look through the log you've attached to AMQ-1445 - but I > think i'll need some more information to successfully resolve whats > going on. Obviously there's an attempt to retrieve a message that's > already been deleted - I just need to find out why ;) > > Could I trouble you to try and devise a junit test case to reproduce? > > thanks, > > Rob > On Oct 5, 2007, at 7:37 AM, David Sitsky wrote: > >> Hi Rob, >> >> I think I spoke too soon. While using AMQ store fixed the initial >> issue I was dealing with, it seems to have other issues in my >> application, where I see the following message before everything hangs: >> >> RecoveryListenerAdapter - Message id ID... could not be recovered from >> the data store! >> >> I created a JIRA record which contains the detailed debug message logs: >> >> https://issues.apache.org/activemq/browse/AMQ-1445 >> >> FWIW, JDBC persistence works fine. Any ideas what might be wrong? >> >> Cheers, >> David >> >> David Sitsky wrote: >>> Hi Rob, >>> Looks like the AMQ store works correctly - thanks for the advice. >>> Cheers, >>> David >>> Rob Davies wrote: >>>> Do you get the same using AMQ store (the default in 5.0) - its the >>>> prefered store over kaha in ActiveMQ 5.0 >>>> >>>> >>>> >>>> cheers, >>>> >>>> Rob >>>> >>>> http://rajdavies.blogspot.com/ >>>> >>>> >>>> >>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote: >>>> >>>>> I have discovered an issue with the latest activemq 5.0 (revision >>>>> 581510) which seems to have been present for some time, at least >>>>> for the last month. >>>>> >>>>> I have an application using transactions, and kaha persistence, and >>>>> two consumers feeding off a queue. >>>>> >>>>> I sometimes see the same message being sent to two consumers. All >>>>> transactions were committed (no rollbacks), and as you can see by >>>>> the debug dump, the message has the same message-ID, but was >>>>> delivered to two separate consumers, running on separate threads, >>>>> using sync receives with a prefetch of 0, with separate sessions of >>>>> course. >>>>> >>>>> Is this a known problem? It doesn't always happen - seems to be a >>>>> race condition. If I turn off persistence, I don't seem to see it. >>>>> >>>>> Here is the message being delivered to consumer 1: >>>>> >>>>> 2007-10-03 17:21:37,310 [Worker 060bacd87f4a40a4867113e2e3e553dd] >>>>> 85475 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message: >>>>> MessageDispatch {commandId = 0, responseRequired = false, >>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, destination = >>>>> queue://worker-items, message = ActiveMQObjectMessage {commandId = >>>>> 50, responseRequired = false, messageId = >>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = >>>>> null, originalTransactionId = null, producerId = >>>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = >>>>> queue://worker-items, transactionId = >>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp >>>>> = 1191396097286, arrival = 0, brokerInTime = 1191396097290, >>>>> brokerOutTime = 1191396097303, correlationId = null, replyTo = >>>>> null, persistent = true, type = null, priority = 4, groupID = null, >>>>> groupSequence = 0, targetConsumerId = null, compressed = false, >>>>> userID = null, content = >>>>> org.apache.activemq.util.ByteSequence@fa8cbc, marshalledProperties >>>>> = null, dataStructure = null, redeliveryCounter = 0, size = 0, >>>>> properties = null, readOnlyProperties = true, readOnlyBody = true, >>>>> droppable = false}, redeliveryCounter = 0} >>>>> >>>>> Here is the same message (check the message ID) being delivered to >>>>> consumer 2: >>>>> >>>>> 2007-10-03 17:21:37,422 [Worker 2c3169d1726b4781a423a239f65c7bd1] >>>>> 85587 DEBUG org.apache.activemq.ActiveMQMessageConsumer - >>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message: >>>>> MessageDispatch {commandId = 0, responseRequired = false, >>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, destination = >>>>> queue://worker-items, message = ActiveMQObjectMessage {commandId = >>>>> 50, responseRequired = false, messageId = >>>>> ID:tamarama-62436-1191396093006-2:2:1:3:1, originalDestination = >>>>> null, originalTransactionId = null, producerId = >>>>> ID:tamarama-62436-1191396093006-2:2:1:3, destination = >>>>> queue://worker-items, transactionId = >>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, timestamp >>>>> = 1191396097286, arrival = 0, brokerInTime = 1191396097290, >>>>> brokerOutTime = 1191396097417, correlationId = null, replyTo = >>>>> null, persistent = true, type = null, priority = 4, groupID = null, >>>>> groupSequence = 0, targetConsumerId = null, compressed = false, >>>>> userID = null, content = >>>>> org.apache.activemq.util.ByteSequence@1a881cc, marshalledProperties >>>>> = null, dataStructure = null, redeliveryCounter = 0, size = 0, >>>>> properties = null, readOnlyProperties = true, readOnlyBody = true, >>>>> droppable = false}, redeliveryCounter = 0} >>>>> >>>>> I'll need to do more investigations tomorrow, but if anybody has >>>>> any ideas, I'd appreciate some insights. >>>>> >>>>> -- >>>>> Cheers, >>>>> David >>>>> >>>>> Nuix Pty Ltd >>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 >>>>> 0699 >>>>> Web: http://www.nuix.com Fax: +61 2 9212 >>>>> 6902 >> >> >> -- >> Cheers, >> David >> >> Nuix Pty Ltd >> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699 >> Web: http://www.nuix.com Fax: +61 2 9212 6902 -- Cheers, David Nuix Pty Ltd Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699 Web: http://www.nuix.com Fax: +61 2 9212 6902 |
|
|
Re: RecoveryListenerAdapter - Message id ID... could not be recovered from the data store: WAS: [Duplicate message delivered to consumers on queue with kaha persistence]
by rajdavies
::
Rate this Message:
Hi David,
I had a test case that periodically could reproduce this - it now can't after ensuring references are always freshed in the Kaha reference store part of AMQ Store. Is your application now fixed? thanks, Rob On Oct 8, 2007, at 2:48 AM, David Sitsky wrote: > Hi Rob, > > Looks like you have fixed this issue - many thanks. > > Cheers, > David > > Rob Davies wrote: >> Hi David, >> I've had a look through the log you've attached to AMQ-1445 - but >> I think i'll need some more information to successfully resolve >> whats going on. Obviously there's an attempt to retrieve a message >> that's already been deleted - I just need to find out why ;) >> Could I trouble you to try and devise a junit test case to reproduce? >> thanks, >> Rob >> On Oct 5, 2007, at 7:37 AM, David Sitsky wrote: >>> Hi Rob, >>> >>> I think I spoke too soon. While using AMQ store fixed the >>> initial issue I was dealing with, it seems to have other issues >>> in my application, where I see the following message before >>> everything hangs: >>> >>> RecoveryListenerAdapter - Message id ID... could not be recovered >>> from the data store! >>> >>> I created a JIRA record which contains the detailed debug message >>> logs: >>> >>> https://issues.apache.org/activemq/browse/AMQ-1445 >>> >>> FWIW, JDBC persistence works fine. Any ideas what might be wrong? >>> >>> Cheers, >>> David >>> >>> David Sitsky wrote: >>>> Hi Rob, >>>> Looks like the AMQ store works correctly - thanks for the advice. >>>> Cheers, >>>> David >>>> Rob Davies wrote: >>>>> Do you get the same using AMQ store (the default in 5.0) - its >>>>> the prefered store over kaha in ActiveMQ 5.0 >>>>> >>>>> >>>>> >>>>> cheers, >>>>> >>>>> Rob >>>>> >>>>> http://rajdavies.blogspot.com/ >>>>> >>>>> >>>>> >>>>> On Oct 3, 2007, at 8:50 AM, David Sitsky wrote: >>>>> >>>>>> I have discovered an issue with the latest activemq 5.0 >>>>>> (revision 581510) which seems to have been present for some >>>>>> time, at least for the last month. >>>>>> >>>>>> I have an application using transactions, and kaha >>>>>> persistence, and two consumers feeding off a queue. >>>>>> >>>>>> I sometimes see the same message being sent to two consumers. >>>>>> All transactions were committed (no rollbacks), and as you can >>>>>> see by the debug dump, the message has the same message-ID, >>>>>> but was delivered to two separate consumers, running on >>>>>> separate threads, using sync receives with a prefetch of 0, >>>>>> with separate sessions of course. >>>>>> >>>>>> Is this a known problem? It doesn't always happen - seems to >>>>>> be a race condition. If I turn off persistence, I don't seem >>>>>> to see it. >>>>>> >>>>>> Here is the message being delivered to consumer 1: >>>>>> >>>>>> 2007-10-03 17:21:37,310 [Worker >>>>>> 060bacd87f4a40a4867113e2e3e553dd] 85475 DEBUG >>>>>> org.apache.activemq.ActiveMQMessageConsumer - >>>>>> ID:tamarama-62436-1191396093006-2:2:3:2 received message: >>>>>> MessageDispatch {commandId = 0, responseRequired = false, >>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:3:2, >>>>>> destination = queue://worker-items, message = >>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired = >>>>>> false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1, >>>>>> originalDestination = null, originalTransactionId = null, >>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3, >>>>>> destination = queue://worker-items, transactionId = >>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, >>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime = >>>>>> 1191396097290, brokerOutTime = 1191396097303, correlationId = >>>>>> null, replyTo = null, persistent = true, type = null, priority >>>>>> = 4, groupID = null, groupSequence = 0, targetConsumerId = >>>>>> null, compressed = false, userID = null, content = >>>>>> org.apache.activemq.util.ByteSequence@fa8cbc, >>>>>> marshalledProperties = null, dataStructure = null, >>>>>> redeliveryCounter = 0, size = 0, properties = null, >>>>>> readOnlyProperties = true, readOnlyBody = true, droppable = >>>>>> false}, redeliveryCounter = 0} >>>>>> >>>>>> Here is the same message (check the message ID) being >>>>>> delivered to consumer 2: >>>>>> >>>>>> 2007-10-03 17:21:37,422 [Worker >>>>>> 2c3169d1726b4781a423a239f65c7bd1] 85587 DEBUG >>>>>> org.apache.activemq.ActiveMQMessageConsumer - >>>>>> ID:tamarama-62436-1191396093006-2:2:1:2 received message: >>>>>> MessageDispatch {commandId = 0, responseRequired = false, >>>>>> consumerId = ID:tamarama-62436-1191396093006-2:2:1:2, >>>>>> destination = queue://worker-items, message = >>>>>> ActiveMQObjectMessage {commandId = 50, responseRequired = >>>>>> false, messageId = ID:tamarama-62436-1191396093006-2:2:1:3:1, >>>>>> originalDestination = null, originalTransactionId = null, >>>>>> producerId = ID:tamarama-62436-1191396093006-2:2:1:3, >>>>>> destination = queue://worker-items, transactionId = >>>>>> TX:ID:tamarama-62436-1191396093006-2:2:5, expiration = 0, >>>>>> timestamp = 1191396097286, arrival = 0, brokerInTime = >>>>>> 1191396097290, brokerOutTime = 1191396097417, correlationId = >>>>>> null, replyTo = null, persistent = true, type = null, priority >>>>>> = 4, groupID = null, groupSequence = 0, targetConsumerId = >>>>>> null, compressed = false, userID = null, content = >>>>>> org.apache.activemq.util.ByteSequence@1a881cc, >>>>>> marshalledProperties = null, dataStructure = null, &n |
