[Python] huggingface model inference: ERROR: Flag 'minloglevel' was defined more than once (in...

I'm tyring to use llama 3.1 70b from huggingface(end goal is to quantize it and deploy in amazon sagemaker), but I'm facing:

ERROR: Flag 'minloglevel' was defined more than once (in files 'src/error.cc' and 'home/conda/feedstock_root/build_artifacts/abseil-split_1720857154496/work/absl/log/flags.cc').
​

The code is super simple(quantization not applied yet) but I cannot make it work(originally I was using a notebook and the kernel died, after exporting the code to python file I found the error mentioned previously).

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer
model_id = "meta-llama/Llama-3.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto"
)
input_text = """<|begin_of_text|><|start_header_id|>user<|end_header_id|>Translate the following English text to French:
'Hello, how are you?'<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
print(input_ids)
output = model.generate(**input_ids, max_new_tokens=10)


I have tried both the 70b and 8b versions but none worked(so I know it is not something specific to 70), I have a feel that this is related to package conflicts or similar and not necesarilly the models. Any hints?

Continue reading...