Repeated Lookup error eventhough NLTK is downloaded: Resource [93mpunkt_tab[0m not found. Please use the NLTK Downloader to obtain the resource: 31m>>> import nltk nltk.download('punkt_tab') Attempted to load [93mtokenizers/punkt_tab/english/[0m Error is found when debugging on line 49. import pandas as pd import nltk import spacy from nltk.tokenize import word_tokenize from nltk.corpus import stopwords from nltk.stem import WordNetLemmatizer import re nltk.download('punkt') nltk.download('stopwords') nltk.download('wordnet') # Set explicit NLTK path nltk.data.path.append("C:\\Users\\Ellie\\nltk_data") # Correct file path file_path = r"C:\Users\Ellie\OneDrive\Documents\Dataset\judge-1377884607_tweet_product_company.csv" # Read the CSV file df = pd.read_csv(file_path, encoding="ISO-8859-1") # Specify encoding if needed # Display first few rows df.head() # Check for missing values df.isnull().sum() #dataset info df.info() # Check unique sentiment values df["is_there_an_emotion_directed_at_a_brand_or_product"].value_counts() # Load spaCy English model nlp = spacy.load("en_core_web_sm") lemmatizer = WordNetLemmatizer() stop_words = set(stopwords.words("english")) def preprocess_text(text): # Lowercase the text text = text.lower() # Remove special characters (keep only letters and spaces) text = re.sub(r'[^a-zA-Z\s]', '', text) # Tokenize the text tokens = word_tokenize(text) # Remove stopwords and words with less than 3 characters, then lemmatize tokens = [lemmatizer.lemmatize(word) for word in tokens if word not in stop_words and len(word) > 2] # Return the cleaned text as a single string return " ".join(tokens) # Apply preprocessing to the tweet text column df["cleaned_tweet"] = df["tweet_text"].astype(str).apply(preprocess_text) # Display sample results df[["tweet_text", "cleaned_tweet"]].head() from sklearn.feature_extraction.text import TfidfVectorizer # Initialize the TF-IDF Vectorizer (limit features for efficiency) vectorizer = TfidfVectorizer(max_features=5000) # Transform cleaned tweets into numerical features X = vectorizer.fit_transform(df["cleaned_tweet"]) # Display the shape of the feature matrix print("TF-IDF matrix shape:", X.shape) script is attempting to tokenize text using word_tokenize(text), which requires the punkt tokenizer. nltk.download('punkt') should ensure the required tokenizer data is downloaded before use. word_tokenize(text) should then successfully split the text into tokens/words. the error message indicates that the necessary punkt tokenizer file is missing. The system attempts to load the resource but fails to find it in the expected location. The error suggests downloading the resource manually using nltk.download('punkt'), but despite this, the issue persists. Continue reading...
