Catheters are usually used to deliver drugs and medications close to the heart and to monitor the vital organs around the chest region for patients who undertook critical surgery. Radiologists often check for the presence of catheters, puncture-needles, guiding sheaths, and various other tube-like structures in interventional radiology. The clinical analysis of X-ray requires a manual pixel-wise annotation which is an excruciating process. In order to address this issue, we attempt to auto-annotate the CXRs using a Self-Supervised Learning approach. Further, the classification task on the catheter is performed based on semantic and perceptual clues (object shapes, colors, and their interactions) of color and class distributions. A generative adversarial network is utilized to learn a mapping to annotate (colorize and identify end-tip points) and classify the given grayscale CXR. The additional number of classes, custom loss function, and attention heads introduced in the model is a unique attempt to ensure robust results in the radiological inferences. It is evident that the qualitative and quantitative results of annotation and classification are viable which resembles how humans perceive such problems. The results are consistent and outperform’s the state-of-the-art supervised learning models in terms of metrics and inference durations. The model being end-to-end in nature, can be integrated along with the existing in-hospital pipeline and will be ready to use instantly.