logo
down
shadow

Get last modified date of a Kafka topic


Get last modified date of a Kafka topic

By : user2185743
Date : November 22 2020, 04:01 AM
This might help you You can't get the timestamp straight forward from the script. Instead you can see the timestamp using the console-consumer script. It shows the CreateTime for a message As mentioned by @Sreekiran also, use the property "print.timestamp" as below :
code :
ncsv@ubuntu:/data/softwares/kafka_2.11-2.0.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testJsonTopic1 --property print.timestamp=true --from-beginning
CreateTime:1544445041225        {"id": "9","name": "wis"}
CreateTime:1544518616282        {"id": "19","name": "wes"}
CreateTime:1544520723808        {"id": "1","name": "test1"}
CreateTime:1544446690508        {"id": "9","name": "wes"}
CreateTime:1544445375123        {"id": "9","name": "wes"}
CreateTime:1544520388080        {"id": "20","name": "test1"}


Share : facebook icon twitter icon
How to find out the latest offset of a Kafka topic to know when my reader is up-to-date with topic?

How to find out the latest offset of a Kafka topic to know when my reader is up-to-date with topic?


By : Paul
Date : March 29 2020, 07:55 AM
I hope this helps . Assuming you are using High level consumer.
High watermark is not available in High level consumer.
code :
**As you mentioned: all the messages in the topic at the moment of start is considered up-to-date**
  AA. Get the partition && offset of the message
  BB. if( offsetMap.get(partition)<=offset) stop Reading from this steam
Replicating messages from one Kafka topic to another kafka topic

Replicating messages from one Kafka topic to another kafka topic


By : Zhenyu Wu
Date : March 29 2020, 07:55 AM
With these it helps If you want to replicate data from one cluster to another then there is one kafka tool called MirrorMaker.
code :
bin/kafka-run-class.sh kafka.tools.MirrorMaker
       --consumer.config consumer.properties
       --producer.config producer.properties --whitelist my-topic
Message pre-processing (topic - topic) - Kafka Connect API vs. Streams vs Kafka Consumer?

Message pre-processing (topic - topic) - Kafka Connect API vs. Streams vs Kafka Consumer?


By : Mick Wang
Date : March 29 2020, 07:55 AM
seems to work fine
provides a lot of good things out of the box (config management, offset storage, error handling, etc.)
code :
bin/kafka-mirror-maker.sh

...

--message.handler <String: A custom      Message handler which will process
  message handler of type                  every record in-between consumer and
  MirrorMakerMessageHandler>               producer.
--message.handler.args <String:          Arguments used by custom message
  Arguments passed to message handler      handler for mirror maker.
  constructor.>
Kafka topic not created automatically on remote kafka after spring boot start(and create on local kafka server)

Kafka topic not created automatically on remote kafka after spring boot start(and create on local kafka server)


By : ladyT
Date : March 29 2020, 07:55 AM
fixed the issue. Will look into that further KafkaAdmin is the kafka spring object that looks for NewTopic objects in your spring context and creates them. If you do not have a KafkaAdmin no creation will take place. You can explicitly create KafkaAdmin (as you show in your code snippet) or indirectly order its creation via the spring kafka configuration properties.
KafkaAdmin is a nice to have it is not related to production or consumption to/ from topics for your application code.
code :
spring:
  kafka:
    bootstrap-servers: remote:9092
2019-03-21 09:18:18.354  INFO 58301 --- [           main] o.a.k.clients.admin.AdminClientConfig    
        : AdminClientConfig values: 
    bootstrap.servers = [remote:9092]
    ...
how to modify message of one kafka topic and send to another kafka topic using java?

how to modify message of one kafka topic and send to another kafka topic using java?


By : Jack Yin
Date : March 29 2020, 07:55 AM
this will help There are many options how you can to this.
Kafka's Streams API is one possibility with the advantage that you don't add a new dependency to your project. As simple stream.map(...).to(...) as program should do this. Check out the docs and examples for more details:
shadow
Privacy Policy - Terms - Contact Us © bighow.org