[SQL] How to select all columns except 2 of them from a large table on pyspark sql?

In joining two tables, I would like to select all columns except 2 of them from a large table with many columns on pyspark sql on databricks.

My pyspark sql:

%sql
set hive.support.quoted.identifiers=none;
select a.*, '?!(b.year|b.month)$).+'
from MY_TABLE_A as a
left join
MY_TABLE_B as b
on a.year = b.year and a.month = b.month


I followed hive:select all column exclude two Hive How to select all but one column?

but, it does not work for me. All columns are in the results. I would like to remove the duplicated columns (year and month in the result).

thanks

Continue reading...